Towards gestural understanding for intelligent robots

Fritsch JN. Towards gestural understanding for intelligent robots. Bielefeld: Universität Bielefeld; 2012.A strong driving force of scientific progress in the technical sciences is the quest for systems that assist humans in their daily life and make their life easier and more enjoyable. Nowadays smartphones are probably the most typical instances of such systems. Another class of systems that is getting increasing attention are intelligent robots. Instead of offering a smartphone touch screen to select actions, these systems are intended to offer a more natural human-machine interface to their users. Out of the large range of actions performed by humans, gestures performed with the hands play a very important role especially when humans interact with their direct surrounding like, e.g., pointing to an object or manipulating it. Consequently, a robot has to understand such gestures to offer an intuitive interface. Gestural understanding is, therefore, a key capability on the way to intelligent robots. This book deals with vision-based approaches for gestural understanding. Over the past two decades, this has been an intensive field of research which has resulted in a variety of algorithms to analyze human hand motions. Following a categorization of different gesture types and a review of other sensing techniques, the design of vision systems that achieve hand gesture understanding for intelligent robots is analyzed. For each of the individual algorithmic steps – hand detection, hand tracking, and trajectory-based gesture recognition – a separate Chapter introduces common techniques and algorithms and provides example methods. The resulting recognition algorithms are considering gestures in isolation and are often not sufficient for interacting with a robot who can only understand such gestures when incorporating the context like, e.g., what object was pointed at or manipulated. Going beyond a purely trajectory-based gesture recognition by incorporating context is an important prerequisite to achieve gesture understanding and is addressed explicitly in a separate Chapter of this book. Two types of context, user-provided context and situational context, are reviewed and existing approaches to incorporate context for gestural understanding are reviewed. Example approaches for both context types provide a deeper algorithmic insight into this field of research. An overview of recent robots capable of gesture recognition and understanding summarizes the currently realized human-robot interaction quality. The approaches for gesture understanding covered in this book are manually designed while humans learn to recognize gestures automatically during growing up. Promising research targeted at analyzing developmental learning in children in order to mimic this capability in technical systems is highlighted in the last Chapter completing this book as this research direction may be highly influential for creating future gesture understanding systems

Vision-based recognition of gestures with context

Fritsch JN. Vision-based recognition of gestures with context. Bielefeld (Germany): Bielefeld University; 2003.Out of the large range of human actions, gestures performed with the hands play a very important role during everyday life. Therefore, their automatic recognition is highly relevant for constructing userfriendly human-machine interfaces. This dissertation presents a new approach to the recognition of manipulative gestures that interact with objects in the environment. The proposed gesture recognition can serve to realize pro-active human-machine interfaces enabling technical systems to observe humans acting in their environment and to react appropriately. For observing the motions of natural hands, a non-intrusive vision-based technique is required. For this purpose, an adaptive skin color segmentation approach that is capable of detecting skin-colored hands in a wide range of lighting conditions is developed. The adaptation step is controlled by using additional scene information to restrict updating of the color model to image areas that actually contain skin areas. With the trajectory data that results from detecting hands in a sequence of images, gesture recognition can be performed. To recognize gestures with context, a new method for incorporating additional scene information in the recognition process is described. The context of a gesture model consists of the current state of the hand and the object that is manipulated. The current hand state is needed to capture the applicability of a gesture model while the manipulated object needs to be present in the vicinity of the hand to enable recognizing the gesture model. Through the proposed context integration, the developed recognition system allows to observe gestures that are mainly characterized by their interaction with the environment and do not have a characteristic trajectory. The performance of the gesture recognition approach is demonstrated with gestures performed in an assembly construction scenario and in a typical office environment. The use of the recognition results for improving human-machine interfaces is shown by applying the gesture recognition in a 'situated artifical communicator' system that is situated in an assembly construction domain. Here the information about the executed gesture can be used for improving dialog interaction by providing information about the hand contents. Besides this direct improvement of the human-machine interface, the recognized gestures can also be used as context knowledge for other system components. This is demonstrated with the observation of construction gestures that provide relevant context information for the vision algorithms aiming at recognizing the objects and assemblies in the construction scenario. In this way the gesture recognition results improve the human-machine interface also indirectly